{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "source": [
    "# 第5章 深度学习计算"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.1 层和块"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 练习 5.1.1 \n",
    "\n",
    "如果将MySequential中存储块的方式更改为Python列表，会出现什么样的问题？ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**解答：** "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;如果将MySequential中存储块的方式从OrderedDict更改为Python列表，代码可以正常计算，但并没有注册给Module。无法像`_modules`一样使用很多内置方完成已经实现的功能。如无法通过`net.state_dict()`访问模型的网络结构和参数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "import torch\n",
    "\n",
    "class MySequential(nn.Module):\n",
    "    # 使用OrderedDict存储块的类\n",
    "    def __init__(self, *args):\n",
    "        super().__init__()\n",
    "        for idx, module in enumerate(args):\n",
    "            # 这里，module是Module子类的一个实例。我们把它保存在'Module'类的成员\n",
    "            # 变量_modules中。_module的类型是OrderedDict\n",
    "            self._modules[str(idx)] = module\n",
    "\n",
    "    def forward(self, X):\n",
    "        # OrderedDict保证了按照成员添加的顺序遍历它们\n",
    "        for block in self._modules.values():\n",
    "            X = block(X)\n",
    "        return X\n",
    "\n",
    "class MySequential_list(nn.Module):\n",
    "    # 使用list存储块的类\n",
    "    def __init__(self, *args):\n",
    "        super(MySequential_list, self).__init__()\n",
    "        self.sequential = []\n",
    "        for module in args:\n",
    "            self.sequential.append(module)\n",
    "\n",
    "    def forward(self, X):\n",
    "        for module in self.sequential:\n",
    "            X = module(X)\n",
    "        return X\n",
    "\n",
    "\n",
    "X = torch.rand(1,10)\n",
    "# 初始化两个block，确保传给MySequential和MySequential_list的是一样的参数。消除nn.Linear初始化时生成随机参数对结果的影响。\n",
    "block1 = nn.Linear(10, 20)\n",
    "block2 = nn.Linear(20, 10)\n",
    "net = MySequential(block1, nn.ReLU(), block2)\n",
    "net_list = MySequential_list(block1, nn.ReLU(), block2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;对比两种方式的结果，可以发现输出完全一致。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "采用python OrderedDict的方式存储 :\n",
      "tensor([[-0.0229,  0.1574, -0.0762, -0.0351, -0.1542,  0.0756,  0.3321,  0.0483,\n",
      "          0.1025,  0.0701]], grad_fn=<AddmmBackward0>)\n",
      "\n",
      "\n",
      "采用python列表的方式存储 :\n",
      "tensor([[-0.0229,  0.1574, -0.0762, -0.0351, -0.1542,  0.0756,  0.3321,  0.0483,\n",
      "          0.1025,  0.0701]], grad_fn=<AddmmBackward0>)\n",
      "\n",
      "\n",
      "两种方式的计算结果是否一致： True\n"
     ]
    }
   ],
   "source": [
    "net_y = net(X)\n",
    "net_list_y = net_list(X)\n",
    "\n",
    "print(\"采用python OrderedDict的方式存储 :\", net_y, sep='\\n')\n",
    "print(\"\\n\")\n",
    "print(\"采用python列表的方式存储 :\", net_list_y, sep='\\n')\n",
    "print(\"\\n\")\n",
    "print(\"两种方式的计算结果是否一致：\", net_y.equal(net_list_y))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;但是在查看模型结构和参数时存在差异。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "采用python OrderedDict的方式存储 :\n",
      "\n",
      "网络结构 : \n",
      " MySequential(\n",
      "  (0): Linear(in_features=10, out_features=20, bias=True)\n",
      "  (1): ReLU()\n",
      "  (2): Linear(in_features=20, out_features=10, bias=True)\n",
      ")\n",
      "参数 ：\n",
      "0.weight tensor([[ 0.2486,  0.2030, -0.2670, -0.2879, -0.2744,  0.2107, -0.2020,  0.3098,\n",
      "          0.2117, -0.2435],\n",
      "        [ 0.0545,  0.0089,  0.2751,  0.1598,  0.1511,  0.3152,  0.1317,  0.0343,\n",
      "         -0.0223,  0.1551]])\n",
      "0.bias tensor([-0.0539, -0.1176])\n",
      "2.weight tensor([[ 0.0448,  0.0088,  0.1115, -0.0057, -0.0651, -0.2142, -0.1881, -0.1496,\n",
      "          0.1201, -0.0022,  0.0889, -0.1150,  0.0061, -0.0892,  0.0075,  0.0689,\n",
      "          0.0851, -0.0310, -0.1352,  0.1313],\n",
      "        [-0.0616,  0.1869,  0.0965,  0.0214,  0.0470, -0.1053,  0.0242, -0.0109,\n",
      "          0.1899, -0.0077,  0.1577,  0.0373,  0.0298, -0.1126,  0.1256, -0.1225,\n",
      "         -0.0995,  0.1725,  0.0678, -0.1200]])\n",
      "2.bias tensor([-0.0475,  0.0316])\n",
      "\n",
      "\n",
      "采用python列表的方式存储 :\n",
      "\n",
      "网络结构 : \n",
      " MySequential(\n",
      "  (0): Linear(in_features=10, out_features=20, bias=True)\n",
      "  (1): ReLU()\n",
      "  (2): Linear(in_features=20, out_features=10, bias=True)\n",
      ")\n",
      "参数： OrderedDict()\n"
     ]
    }
   ],
   "source": [
    "# 使用_modules方便打印net的网络结构和参数，而list则无法做到\n",
    "print(\"采用python OrderedDict的方式存储 :\\n\")\n",
    "print(\"网络结构 : \\n\", net)\n",
    "# 为了可视化效果简洁，这里只展示了每个参数的部分值\n",
    "print(\"参数 ：\", \"\\n\".join([str(k) + str(\" \") + str(v[:2]) for k,v in net.state_dict().items()]), sep='\\n')\n",
    "print(\"\\n\")\n",
    "print(\"采用python列表的方式存储 :\\n\")\n",
    "print(\"网络结构 : \\n\", net) \n",
    "print(\"参数：\", net_list.state_dict())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 练习 5.1.2\n",
    "\n",
    "实现一个块，它以两个块为参数，例如net1和net2，并返回前向传播中两个网络的并联输出。这也被称为并行块。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**解答：**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;在本书7.4节中`GoogleLet`模型中的`Inception`块使用了并行块技术。\n",
    "下面代码实现了一个并行网络，由两个子网络组成。输入数据先分别经过两个子网络的计算，分别得到两个部分的输出结果，然后在通道维度上合并结果得到最终输出。\n",
    "\n",
    "&emsp;&emsp;其中，`net1`和`net2`分别表示两个子网络，`torch.cat`在指定维度上拼接张量。输出结果的大小为`torch.Size([2, 36])`，其中第一个维度表示batch_size为2，第二个维度表示输出特征图的通道数为36，正好为两个模型的输出特征图通道数之和（12+24=36）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "模型结构 :\n",
      "Parallel(\n",
      "  (net1): Sequential(\n",
      "    (0): Linear(in_features=10, out_features=12, bias=True)\n",
      "    (1): ReLU()\n",
      "  )\n",
      "  (net2): Sequential(\n",
      "    (0): Linear(in_features=10, out_features=24, bias=True)\n",
      "    (1): ReLU()\n",
      "  )\n",
      ")\n",
      "模型输出的形状 : torch.Size([2, 36])\n"
     ]
    }
   ],
   "source": [
    "import torch.nn as nn\n",
    "import torch\n",
    "\n",
    "\n",
    "class Parallel(nn.Module):\n",
    "    # 自定义并行块\n",
    "    def __init__(self, net1, net2):\n",
    "        super().__init__()\n",
    "        self.net1=net1 # 第一个子网络\n",
    "        self.net2=net2 # 第二个子网络\n",
    "        \n",
    "    def forward(self,X):\n",
    "        x1= self.net1(X) # 第一个子网络的输出\n",
    "        x2= self.net2(X) # 第二个子网络的输出\n",
    "        return torch.cat((x1,x2),dim=1) # 在通道维度上合并输出结果\n",
    "      \n",
    "X = torch.rand(2,10) # 输入数据\n",
    "net = Parallel(nn.Sequential(nn.Linear(10,12),nn.ReLU()), nn.Sequential(nn.Linear(10,24),nn.ReLU())) # 实例化并行网络\n",
    "output = net(X)\n",
    "print(\"模型结构 :\", net, sep=\"\\n\") # 查看模型结构\n",
    "print(\"模型输出的形状 :\", output.size()) # 输出结果的大小"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 练习 5.1.3\n",
    "\n",
    "假设我们想要连接同一网络的多个实例。实现一个函数，该函数生成同一个块的多个实例，并在此基础上构建更大的网络。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**解答：** "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;下面代码定义了一个函数`create_network`，该函数接受四个参数：`num_instances`、`input_size`、`hidden_size` 和 `output_size`，并返回一个Sequential模型。\n",
    "\n",
    "&emsp;&emsp;其中，该网络模型首先包含 `num_instances` 个相同的线性层，每个线性层有两个子层：一个输入维度为 `input_size`，输出维度为 `hidden_size` 的全连接层，和一个 ReLU 非线性激活层。然后，这 `hidden_size` 个线性层连接在一起作为整个网络的前馈部分。最后，额外添加一个输出层，其输入维度为 `input_size`，输出维度为 `output_size`。\n",
    "\n",
    "&emsp;&emsp;因此，最终的网络结构是由 `output_size` 个相同的线性层组成的前馈神经网络，每个线性层内部包含一个全连接层和一个ReLU激活层，以及一个独立的输出层。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sequential(\n",
      "  (0): Sequential(\n",
      "    (0): Linear(in_features=10, out_features=5, bias=True)\n",
      "    (1): ReLU()\n",
      "    (2): Linear(in_features=5, out_features=10, bias=True)\n",
      "  )\n",
      "  (1): Sequential(\n",
      "    (0): Linear(in_features=10, out_features=5, bias=True)\n",
      "    (1): ReLU()\n",
      "    (2): Linear(in_features=5, out_features=10, bias=True)\n",
      "  )\n",
      "  (2): Sequential(\n",
      "    (0): Linear(in_features=10, out_features=5, bias=True)\n",
      "    (1): ReLU()\n",
      "    (2): Linear(in_features=5, out_features=10, bias=True)\n",
      "  )\n",
      "  (output): Linear(in_features=10, out_features=2, bias=True)\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "import torch.nn as nn\n",
    "import torch\n",
    "\n",
    "\n",
    "def create_network(num_instances, input_size, hidden_size, output_size):\n",
    "    # 创建一个线性层\n",
    "    linear_layer = nn.Sequential(\n",
    "        nn.Linear(input_size, hidden_size), nn.ReLU(),\n",
    "        nn.Linear(hidden_size, input_size)\n",
    "    )\n",
    "    \n",
    "    # 创建多个相同结构的实例并连接\n",
    "    instances = [linear_layer for _ in range(num_instances)]\n",
    "    network = nn.Sequential(*instances)\n",
    "    \n",
    "    # 添加输出层\n",
    "    output_layer = nn.Linear(input_size, output_size)\n",
    "    network.add_module(\"output\", output_layer)\n",
    "    \n",
    "    return network\n",
    "\n",
    "# 模型参数\n",
    "num_instances=3\n",
    "input_size=10\n",
    "hidden_size=5\n",
    "output_size=2\n",
    "\n",
    "# 示例用法\n",
    "net = create_network(num_instances=num_instances, input_size=input_size, \n",
    "                     hidden_size=hidden_size, output_size=output_size) # 重复3次相同的线形层\n",
    "\n",
    "# 查看模型结构\n",
    "print(net)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.2 参数管理 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 练习 5.2.1\n",
    "\n",
    "使用`NestMLP`模型，访问各个层的参数。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " **解答：**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;引用上5.1节中的`NestMLP`模型，可以使用以下代码访问该模型各个层的参数，输出结果将显示每个层对应的参数名称、形状和具体参数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "访问net层的参数\n",
      "参数名称: 0.weight, 形状:torch.Size([64, 20])\n",
      "参数名称: 0.bias, 形状:torch.Size([64])\n",
      "参数名称: 2.weight, 形状:torch.Size([32, 64])\n",
      "参数名称: 2.bias, 形状:torch.Size([32])\n",
      "\n",
      "访问linear层的参数\n",
      "参数名称: weight, 形状:torch.Size([16, 32])\n",
      "参数名称: bias, 形状:torch.Size([16])\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "\n",
    "class NestMLP(nn.Module):\n",
    "    # 5.1节中的NestMLP模型\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.net = nn.Sequential(nn.Linear(20, 64), nn.ReLU(),\n",
    "                                 nn.Linear(64, 32), nn.ReLU())\n",
    "        self.linear = nn.Linear(32, 16)\n",
    "\n",
    "    def forward(self, X):\n",
    "        return self.linear(self.net(X))\n",
    "\n",
    "model = NestMLP()\n",
    "\n",
    "# 访问net层的参数\n",
    "print('访问net层的参数')\n",
    "for name, param in model.net.named_parameters():\n",
    "    print(f\"参数名称: {name}, 形状:{param.shape}\") # 打印具体参数值所占页面空间较大，读者可按照下面注释的代码自行打印查看。\n",
    "    # print(f\"参数名称: {name}, 形状:{param.shape}, 具体参数值: {param}\")\n",
    "\n",
    "# 访问linear层的参数\n",
    "print('\\n访问linear层的参数')\n",
    "for name, param in model.linear.named_parameters():\n",
    "    print(f\"参数名称: {name}, 形状:{param.shape}\") # 打印具体参数值所占页面空间较大，读者可按照下面注释的代码自行打印查看。\n",
    "    # print(f\"参数名称: {name}, 形状:{param.shape}, 具体参数值: {param}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 练习 5.2.2\n",
    "\n",
    "查看初始化模块文档以了解不同的初始化方法。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**解答：**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;通过查看深度学习框架文档，有以下初始化方法 （参考链接：https://pytorch.org/docs/stable/nn.init.html ）\n",
    "- `torch.nn.init.uniform_(tensor, a=0.0, b=1.0)`：以均匀分布$U(a,b)$初始化输入张量。\n",
    "\n",
    "- `torch.nn.init.normal_(tensor, mean=0.0, std=1.0)`：以正态分布$N(mean, std^2)$初始化输入张量。\n",
    "\n",
    "- `torch.nn.init.constant_(tensor, val)`：以一确定数值初始化输入张量。\n",
    "\n",
    "- `torch.nn.init.ones_(tensor)`：用标量值 1 初始化输入张量。\n",
    "\n",
    "- `torch.nn.init.zeros_(tensor)`：用标量值 0 初始化输入张量。\n",
    "\n",
    "- `torch.nn.init.eye_(tensor)`：用单位矩阵初始化二维输入张量。\n",
    "\n",
    "- `torch.nn.init.xavier_uniform_(tensor, gain=1.0)`：从均匀分布$U(−a, a)$中采样，初始化输入张量，其中$a$的值由如下公式确定\n",
    "\n",
    "  $$a= gain * \\sqrt{\\frac{6}{fan_{in}+fan_{out}}}$$\n",
    "  \n",
    "  其中$gain$的取值如下表所示\n",
    "<style> table { margin: auto;} </style>\n",
    "非线性函数 | gain值\n",
    ":----:|:----:\n",
    "Linear/Identity | 1\n",
    "Conv1D | 1\n",
    "Conv2D | 1\n",
    "Conv3D | 1\n",
    "Sigmoid | 1\n",
    "Tanh | $\\displaystyle\\frac{5}{3}$\n",
    "ReLU | $\\sqrt{2}$\n",
    "Leaky ReLU | $$\\sqrt{\\frac{2}{1+negative\\_slope^2}}$$\n",
    "SELU | 1 (adaptive)\n",
    "\n",
    "- `torch.nn.init.xavier_normal_(tensor, gain=1.0)`:从正态分布$N(0,std^2)$中采样，初始化输入张量，其中$std$值由下式确定：\n",
    "\n",
    "  $$std= gain * \\sqrt{\\frac{2}{fan_{in}+fan_{out}}}$$\n",
    "\n",
    "- `torch.nn.init.kaiming_uniform_(tensor, a=0, mode='fan_in', nonlinearity='leaky_relu')`:服从均匀分布$U(−bound, bound)$，其中$bound$值由下式确定\n",
    "\n",
    "  $$bound= gain * \\sqrt{\\frac{3}{fan_{mode}}}$$\n",
    "\n",
    "- `torch.nn.init.kaiming_normal_(tensor, a=0, mode='fan_in', nonlinearity='leaky_relu')`:服从从正态分布$N(0,std^2)$中采样，其中$std$值由下式确定\n",
    "\n",
    "  $$std= \\frac{gain}{\\sqrt{fan_{mode}}}$$\n",
    "  \n",
    "- `torch.nn.init.trunc_normal_(tensor, mean=0.0, std=1.0, a=- 2.0, b=2.0)`:用从截断的正态分布中提取的值初始化输入张量。这些值实际上是从正态分布 $N(mean, std^2)$中提取的。\n",
    "\n",
    "- `torch.nn.init.sparse_(tensor, sparsity, std=0.01)`：将 2D 输入张量初始化为稀疏矩阵，其中非零元素服从正态分布$N(0,0.01)$。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 练习 5.2.3\n",
    "\n",
    "构建包含共享参数层的多层感知机并对其进行训练。在训练过程中，观察模型各层的参数和梯度。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**解答：** "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;在训练过程中，我们每个epoch都打印了每层的参数和梯度。可以看到`shared_fc`层的参数和梯度都是相同的，因为它们共享同一个参数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0\n",
      "参数名称: 0.weight, 具体参数值: tensor([[-0.3173,  0.4538,  0.4591],\n",
      "        [-0.1105, -0.4601, -0.3338]]), 参数梯度: tensor([[-0.0043, -0.0171, -0.0087],\n",
      "        [ 0.0000,  0.0000,  0.0000]])\n",
      "参数名称: 0.bias, 具体参数值: tensor([0.2715, 0.1842]), 参数梯度: tensor([-0.0122,  0.0000])\n",
      "参数名称: 2.weight, 具体参数值: tensor([[-0.1411,  0.3762,  0.2200,  0.4228],\n",
      "        [ 0.0919,  0.3894,  0.2406,  0.3502]]), 参数梯度: tensor([[ 0.0000, -0.0536,  0.0000,  0.0000],\n",
      "        [-0.1513, -0.0059, -0.0000, -0.0047]])\n",
      "参数名称: 2.bias, 具体参数值: tensor([0.1031, 0.4674]), 参数梯度: tensor([-0.0925, -0.1450])\n",
      "参数名称: 6.weight, 具体参数值: tensor([[-0.2783, -0.3934, -0.2389, -0.0807],\n",
      "        [-0.3443, -0.3194,  0.2846,  0.4655]]), 参数梯度: tensor([[-0.1743, -0.3768, -0.0056, -0.0000],\n",
      "        [ 0.2298,  0.4968,  0.0074,  0.0000]])\n",
      "参数名称: 6.bias, 具体参数值: tensor([-0.3407,  0.1918]), 参数梯度: tensor([-0.5447,  0.7183])\n",
      "Epoch: 0, Loss: 0.60952228307724\n",
      "\n",
      "Epoch: 1\n",
      "参数名称: 0.weight, 具体参数值: tensor([[-0.3173,  0.4540,  0.4592],\n",
      "        [-0.1105, -0.4601, -0.3338]]), 参数梯度: tensor([[-0.0044, -0.0176, -0.0089],\n",
      "        [ 0.0000,  0.0000,  0.0000]])\n",
      "参数名称: 0.bias, 具体参数值: tensor([0.2717, 0.1842]), 参数梯度: tensor([-0.0125,  0.0000])\n",
      "参数名称: 2.weight, 具体参数值: tensor([[-0.1411,  0.3767,  0.2200,  0.4228],\n",
      "        [ 0.0935,  0.3895,  0.2406,  0.3503]]), 参数梯度: tensor([[ 0.0000, -0.0546,  0.0000,  0.0000],\n",
      "        [-0.1530, -0.0084, -0.0000, -0.0050]])\n",
      "参数名称: 2.bias, 具体参数值: tensor([0.1040, 0.4690]), 参数梯度: tensor([-0.0935, -0.1507])\n",
      "参数名称: 6.weight, 具体参数值: tensor([[-0.2766, -0.3896, -0.2389, -0.0807],\n",
      "        [-0.3465, -0.3243,  0.2845,  0.4655]]), 参数梯度: tensor([[-0.1740, -0.3746, -0.0025, -0.0000],\n",
      "        [ 0.2285,  0.4921,  0.0033,  0.0000]])\n",
      "参数名称: 6.bias, 具体参数值: tensor([-0.3353,  0.1847]), 参数梯度: tensor([-0.5393,  0.7085])\n",
      "Epoch: 1, Loss: 0.5946049094200134\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "\n",
    "# 模型参数\n",
    "input_size = 3\n",
    "hidden_size = 4\n",
    "output_size = 3\n",
    "lr = 0.01\n",
    "epochs = 2\n",
    "\n",
    "# 构建带有共享参数层的多层感知机\n",
    "shared_fc = nn.Linear(hidden_size, hidden_size)\n",
    "MLP = nn.Sequential(nn.Linear(input_size, hidden_size), nn.ReLU(),\n",
    "                    shared_fc, nn.ReLU(),\n",
    "                    shared_fc, nn.ReLU(),\n",
    "                    nn.Linear(hidden_size, output_size)\n",
    ")\n",
    "\n",
    "# 训练数据\n",
    "X = torch.randn(1, input_size)\n",
    "Y = torch.randn(1, output_size)\n",
    "# 优化器\n",
    "optimizer = optim.SGD(MLP.parameters(), lr=lr)\n",
    "# 训练模型\n",
    "for epoch in range(epochs):\n",
    "    # 前向传播和计算损失\n",
    "    Y_pred = MLP(X)\n",
    "    loss = nn.functional.mse_loss(Y_pred, Y)\n",
    "    # 反向传播和更新梯度\n",
    "    optimizer.zero_grad()\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "    # 打印每层的参数和梯度\n",
    "    print(f'Epoch: {epoch}')\n",
    "    for name, param in MLP.named_parameters():\n",
    "        print(f\"参数名称: {name}, 具体参数值: {param.data[:2]}, 参数梯度: {param.grad[:2]}\") # 为了节省页面空间，这里只打印了每个参数和梯度的前两维，读者可以自行调整查看维度\n",
    "    print(f'Epoch: {epoch}, Loss: {loss.item()}\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看出这个过程只有两个共享层的参数和梯度数值是一样的，我们直接对这两层结果进行进一步的确认。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "参数是否相同: tensor([[True, True, True, True],\n",
      "        [True, True, True, True],\n",
      "        [True, True, True, True],\n",
      "        [True, True, True, True]])\n",
      "参数是否同时变化: tensor([True, True, True, True])\n",
      "是否时同一个对象: True\n",
      "梯度是否相同: tensor([[True, True, True, True],\n",
      "        [True, True, True, True],\n",
      "        [True, True, True, True],\n",
      "        [True, True, True, True]])\n",
      "梯度是否同时变化: tensor([True, True, True, True])\n",
      "是否时同一个对象: True\n"
     ]
    }
   ],
   "source": [
    "# 检查参数是否相同\n",
    "print(f\"参数是否相同: {MLP[2].weight.data == MLP[4].weight.data}\") \n",
    "# 确保它们实际上是同一个对象，而不只是有相同的值\n",
    "MLP[2].weight.data[0, 0] = 100 \n",
    "print(f\"参数是否同时变化: {MLP[2].weight.data[0] == MLP[4].weight.data[0]}\") \n",
    "print(f\"是否时同一个对象: {MLP[2].weight.data.equal(MLP[4].weight.data)}\")\n",
    "\n",
    "\n",
    "# 检查参数是否相同\n",
    "print(f\"梯度是否相同: {MLP[2].weight.grad == MLP[4].weight.grad}\") \n",
    "# 确保它们实际上是同一个对象，而不只是有相同的值\n",
    "MLP[2].weight.grad[0, 0] = 100 \n",
    "print(f\"梯度是否同时变化: {MLP[2].weight.grad[0] == MLP[4].weight.grad[0]}\") \n",
    "print(f\"是否时同一个对象: {MLP[2].weight.grad.equal(MLP[4].weight.grad)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 练习 5.2.4\n",
    "\n",
    "为什么共享参数是个好主意？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**解答：** "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;1. 节约内存：共享参数可以减少模型中需要存储的参数数量，从而减少内存占用。\n",
    "\n",
    "&emsp;&emsp;2. 加速收敛：共享参数可以让模型更加稳定，加速收敛。\n",
    "\n",
    "&emsp;&emsp;3. 提高泛化能力：共享参数可以帮助模型更好地捕捉数据中的共性，提高模型的泛化能力。\n",
    "\n",
    "&emsp;&emsp;4. 加强模型的可解释性：共享参数可以让模型更加简洁明了，加强模型的可解释性。 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.3 延后初始化 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 练习 5.3.1 \n",
    "\n",
    "如果指定了第一层的输入尺寸，但没有指定后续层的尺寸，会发生什么？是否立即进行初始化？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**解答：** "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;如果指定了第一层的输入尺寸，但没有指定后续层的尺寸，可以正常运行。第一层会立即初始化，但其他层是直到数据第一次通过模型传递才会初始化。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--- 数据未通过模型 ---\n",
      "第一层权重： Parameter containing:\n",
      "tensor([[-0.3413, -0.5182, -0.3627],\n",
      "        [ 0.2411, -0.4914,  0.0811],\n",
      "        [ 0.0203,  0.4713,  0.1287],\n",
      "        [-0.1224,  0.2020,  0.0286]], requires_grad=True)\n",
      "其它层的权重： <UninitializedParameter>\n",
      "--- 数据第一次通过模型 ---\n",
      "第一层权重： Parameter containing:\n",
      "tensor([[-0.3413, -0.5182, -0.3627],\n",
      "        [ 0.2411, -0.4914,  0.0811],\n",
      "        [ 0.0203,  0.4713,  0.1287],\n",
      "        [-0.1224,  0.2020,  0.0286]], requires_grad=True)\n",
      "其它层的权重： Parameter containing:\n",
      "tensor([[ 0.0142,  0.1109, -0.2118,  0.2039],\n",
      "        [-0.3138, -0.1928,  0.2217, -0.0317],\n",
      "        [ 0.3973,  0.2929,  0.0405,  0.3447],\n",
      "        [-0.0141,  0.0498, -0.4321, -0.2023],\n",
      "        [ 0.2127,  0.0106, -0.3949, -0.2392],\n",
      "        [-0.3214,  0.4515, -0.0378,  0.2473],\n",
      "        [-0.2836,  0.3446,  0.0734, -0.4093],\n",
      "        [ 0.4049,  0.1875,  0.0278,  0.4894]], requires_grad=True)\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "\n",
    "\"\"\"延后初始化\"\"\"\n",
    "# 定义模型\n",
    "net = nn.Sequential(nn.Linear(3, 4), nn.ReLU(), nn.LazyLinear(8))\n",
    "\n",
    "# 尚未初始化\n",
    "print(\"--- 数据未通过模型 ---\")\n",
    "print(\"第一层权重：\", net[0].weight)\n",
    "print(\"其它层的权重：\", net[2].weight)\n",
    "\n",
    "X = torch.rand(2, 3)\n",
    "net(X)\n",
    "# 已初始化\n",
    "print(\"--- 数据第一次通过模型 ---\")\n",
    "print(\"第一层权重：\", net[0].weight)\n",
    "print(\"其它层的权重：\", net[2].weight)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 练习 5.3.2\n",
    "\n",
    "如果指定了不匹配的维度会发生什么？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**解答：** "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;如果指定了不匹配的维度，会由于矩阵乘法时维度不匹配而报错。在下面的代码中便指定了不匹配的维度。\n",
    "\n",
    "&emsp;&emsp;由于第一层 nn.Linear(20, 256) 的输入维度为 20，所以输入数据 X 的最后一维必须为 20 才能与该层的权重矩阵相乘。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mat1 and mat2 shapes cannot be multiplied (2x10 and 20x256)\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "\n",
    "# 定义模型\n",
    "net = nn.Sequential(\n",
    "    nn.Linear(20, 256), nn.ReLU(),\n",
    "    nn.LazyLinear(128), nn.ReLU(),\n",
    "    nn.LazyLinear(10))\n",
    "\n",
    "X = torch.rand(2, 10)\n",
    "try:\n",
    "    net(X)\n",
    "except Exception as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 练习 5.3.3 \n",
    "\n",
    "如果输入具有不同的维度，需要做什么？提示：查看参数绑定的相关内容。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**解答：**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;如果输入维度比指定维度小，可以考虑使用 padding 填充；如果输入维度比指定维度大，可以考虑用 PCA 等降维方法，将维度降至指定维度。对于不同的维度，还可以添加一个额外的线性层，并将第一个线性层的权重与该层的权重绑定在一起。这样就可以解决维度不匹配的问题，并且保持模型的权重不变。注意，在下面的代码中，我们假设第一个线性层的偏置项为零，因此不需要对其进行参数绑定。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mat1 and mat2 shapes cannot be multiplied (2x10 and 20x256)\n",
      "第一个线性层的维度：torch.Size([256, 20])\n",
      "额外的线性层的维度：torch.Size([256, 10])\n",
      "第一个线性层的新维度：torch.Size([256, 10])\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.0349,  0.0270, -0.0077, -0.0073,  0.0962,  0.0133,  0.1692, -0.1084,\n",
       "          0.0926,  0.0414],\n",
       "        [ 0.0433, -0.0023, -0.0376,  0.0317,  0.0810,  0.0140,  0.1227, -0.0980,\n",
       "          0.0960,  0.0180]], grad_fn=<AddmmBackward0>)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "# 定义模型\n",
    "net = nn.Sequential(\n",
    "    nn.Linear(20, 256), nn.ReLU(),\n",
    "    nn.Linear(256, 128), nn.ReLU(),\n",
    "    nn.Linear(128, 10))\n",
    "\n",
    "X = torch.rand(2, 10)\n",
    "\n",
    "try:\n",
    "    net(X)\n",
    "except Exception as e:\n",
    "    print(e)\n",
    "\n",
    "# 添加额外的线性层\n",
    "extra_layer = nn.Linear(10, 256)\n",
    "print(f\"第一个线性层的维度：{net[0].weight.shape}\")\n",
    "print(f\"额外的线性层的维度：{extra_layer.weight.shape}\")\n",
    "\n",
    "# 将第一个线性层与额外的线性层的权重进行绑定\n",
    "net[0].weight = extra_layer.weight\n",
    "print(f\"第一个线性层的新维度：{net[0].weight.shape}\")\n",
    "\n",
    "# 使用新的输入（维度为20）调用模型\n",
    "net(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.4 自定义层 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 练习 5.4.1 \n",
    "\n",
    "设计一个接受输入并计算张量降维的层，它返回$y_k = \\sum_{i, j} W_{ijk} x_i x_j$。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**解答：** "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;这个公式表示一个线性变换，将输入张量$x$中所有可能的二元组$(x_i,x_j)$进行组合，并对它们进行加权求和。其中，$W_{ijk}$表示权重张量中第$i,j,k$个元素的值。具体而言，该公式计算了输入张量$x$中所有二元组$(x_i, x_j)$对应的特征向量$u_{ij}$：\n",
    "\n",
    "\n",
    "$$\n",
    "u_{ij} = x_i \\cdot x_j\n",
    "$$\n",
    "\n",
    "\n",
    "&emsp;&emsp;然后，根据权重张量$W$中的权重$W_{ijk}$，对所有特征向量$u_{ij}$进行线性组合，得到输出向量$y_k$为：\n",
    "\n",
    "\n",
    "$$\n",
    "y_k = \\sum_{i,j} W_{ijk} u_{ij} = \\sum_{i,j} W_{ijk} x_i x_j\n",
    "$$\n",
    "\n",
    "\n",
    "&emsp;&emsp;该操作可以被视为一种降维操作，将高维输入$x$映射到低维输出空间$y$中。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([2, 5])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "class TensorReduction(nn.Module):\n",
    "    def __init__(self, dim1, dim2):\n",
    "        super(TensorReduction, self).__init__()\n",
    "        # 定义一个可训练的权重参数，维度为(dim2, dim1, dim1)\n",
    "        self.weight = nn.Parameter(torch.rand(dim2, dim1, dim1))\n",
    "\n",
    "    def forward(self, X):\n",
    "        # 初始化一个全零张量，大小为(X.shape[0], self.weight.shape[0])\n",
    "        Y = torch.zeros(X.shape[0], self.weight.shape[0])\n",
    "        for k in range(self.weight.shape[0]):\n",
    "            # 计算temp = X @ weight[k] @ X^T\n",
    "            temp = torch.matmul(X, self.weight[k]) @ X.T\n",
    "            # 取temp的对角线元素，存入Y[:, k]\n",
    "            Y[:, k] = temp.diagonal()\n",
    "        return Y\n",
    "\n",
    "# 创建一个TensorReduction层，dim1=10, dim2=5\n",
    "layer = TensorReduction(10, 5)\n",
    "# 创建一个大小为(2, 10)的张量X\n",
    "X = torch.rand(2, 10)\n",
    "# 对layer(X)进行前向传播，返回一个大小为(2, 5)的张量\n",
    "layer(X).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 练习 5.4.2 \n",
    "\n",
    "设计一个返回输入数据的傅立叶系数前半部分的层。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**解答：** "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;傅里叶级数可以参考[维基百科](https://en.wikipedia.org/wiki/Fourier_series)中的定义。\n",
    "\n",
    "&emsp;&emsp;在`torch`中有相应的函数可以轻松的实现傅里叶级数，如下代码所示："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[ 5.3966+0.0000j,  0.0402+0.3644j],\n",
       "         [ 0.1826+0.0000j, -0.0390+1.0092j]]])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch.nn as nn\n",
    "import torch.fft as fft\n",
    "\n",
    "class FourierLayer(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(FourierLayer, self).__init__()\n",
    "\n",
    "    def forward(self, x):\n",
    "        # 对输入的张量 x 进行快速傅里叶变换\n",
    "        x = fft.fftn(x)\n",
    "        # 取出第三个维度的前半部分，即去掉直流分量和镜像分量\n",
    "        x = x[:, :, :x.shape[2] // 2]\n",
    "        # 返回处理后的张量\n",
    "        return x\n",
    "\n",
    "# 创建一个随机数值为 [0, 1) 的形状为 (1, 2, 5) 的张量 X\n",
    "X = torch.rand(1, 2, 5)\n",
    "# 实例化一个 FourierLayer 的网络对象 net\n",
    "net = FourierLayer()\n",
    "# 将 X 输入到网络 net 中进行前向计算，并输出结果\n",
    "net(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.5 读写文件 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  练习 5.5.1\n",
    "\n",
    "即使不需要将经过训练的模型部署到不同的设备上，存储模型参数还有什么实际的好处？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**解答：** "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;1. 便于模型恢复和微调：存储模型参数可以在训练中断后继续训练，或者在新的数据集上进行微调，而无需从头开始训练。\n",
    "\n",
    "&emsp;&emsp;2. 节省存储空间：相比于保存完整的模型结构，保存模型参数通常占用更少的存储空间，这在处理大型模型或存储空间受限的情况下尤为重要。\n",
    "\n",
    "&emsp;&emsp;3. 便于共享和复现：存储模型参数可以方便地共享和复现已经训练好的模型，其他人可以直接加载这些参数并使用它们进行预测或进一步训练。\n",
    "\n",
    "&emsp;&emsp;4. 便于调试和分析：通过检查和分析模型参数，可以更容易地诊断模型中存在的问题，并对模型进行调整和优化。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;下面将以一个简单的例子具体说明。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;1. 便于模型恢复和微调：假设我们有一个简单的神经网络模型，我们可以使用 PyTorch 保存和加载模型参数："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<All keys matched successfully>"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "# 定义模型\n",
    "class SimpleModel(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(SimpleModel, self).__init__()\n",
    "        self.fc = nn.Linear(10, 2)\n",
    "\n",
    "    def forward(self, x):\n",
    "        return self.fc(x)\n",
    "\n",
    "model = SimpleModel()\n",
    "\n",
    "# 保存模型参数\n",
    "torch.save(model.state_dict(), 'model_parameters.pth')\n",
    "\n",
    "# 加载模型参数进行微调\n",
    "model.load_state_dict(torch.load('model_parameters.pth'))\n",
    "# 继续训练或进行微调..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;2. 节省存储空间：在上述示例中，通过使用 `torch.save(model.state_dict(), 'model_parameters.pth')` 只保存模型参数，而不是整个模型，可以节省存储空间。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;3. 便于共享和复现：保存模型参数后，可以将 `model_parameters.pth` 文件与他人共享。他们可以使用以下代码加载参数并复现模型："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<All keys matched successfully>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = SimpleModel()\n",
    "model.load_state_dict(torch.load('model_parameters.pth'))\n",
    "# 使用模型进行预测或进一步训练..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;4. 便于调试和分析：通过检查模型参数，可以对模型进行调试和分析。例如，可以打印出模型的权重和偏置："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "fc.weight tensor([[ 0.0993, -0.2440, -0.2677,  0.0445,  0.1687, -0.1331, -0.1430,  0.2430,\n",
      "         -0.1612,  0.2804],\n",
      "        [ 0.1476, -0.0100,  0.1797, -0.2724, -0.3085,  0.2196,  0.2533,  0.2434,\n",
      "          0.1726,  0.2166]])\n",
      "fc.bias tensor([ 0.0230, -0.2286])\n"
     ]
    }
   ],
   "source": [
    "for name, param in model.named_parameters():\n",
    "    if param.requires_grad:\n",
    "        print(name, param.data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 练习 5.5.2\n",
    "\n",
    "假设我们只想复用网络的一部分，以将其合并到不同的网络架构中。比如想在一个新的网络中使用之前网络的前两层，该怎么做？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**解答：** "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;使用保存模型某层参数的办法，保存网络的前两层，然后再加载到新的网络中使用。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[True, True, True,  ..., True, True, True],\n",
      "        [True, True, True,  ..., True, True, True],\n",
      "        [True, True, True,  ..., True, True, True],\n",
      "        ...,\n",
      "        [True, True, True,  ..., True, True, True],\n",
      "        [True, True, True,  ..., True, True, True],\n",
      "        [True, True, True,  ..., True, True, True]])\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "from torch.nn import functional as F\n",
    "\n",
    "class MLP(nn.Module):\n",
    "    \"\"\"定义 MLP 类。\"\"\"\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.hidden = nn.Linear(20, 256)  # 定义隐藏层，输入尺寸为 20，输出尺寸为 256。\n",
    "        self.output = nn.Linear(256, 10)  # 定义输出层，输入尺寸为 256，输出尺寸为 10。\n",
    "\n",
    "    def forward(self, x):\n",
    "        \"\"\"定义前向传播函数。\"\"\"\n",
    "        # 使用 ReLU 激活函数，计算隐藏层和输出层的输出。\n",
    "        return self.output(F.relu(self.hidden(x)))\n",
    "\n",
    "class MLP_new(nn.Module):\n",
    "    \"\"\"定义 MLP_new 类。\"\"\"\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.hidden = nn.Linear(20, 256)  # 定义隐藏层，输入尺寸为 20，输出尺寸为 256。\n",
    "        self.output = nn.Linear(256, 10)  # 定义输出层，输入尺寸为 256，输出尺寸为 10。\n",
    "\n",
    "    def forward(self, x):\n",
    "        \"\"\"定义前向传播函数。\"\"\"\n",
    "        # 使用 ReLU 激活函数，计算隐藏层和输出层的输出。\n",
    "        return self.output(F.relu(self.hidden(x)))\n",
    "\n",
    "net = MLP()  # 创建 MLP 的实例。\n",
    "# 将隐藏层的参数保存到文件中。\n",
    "torch.save(net.hidden.state_dict(), 'mlp.hidden.params')\n",
    "\n",
    "clone = MLP_new()  # 创建另一个 MLP 的实例。\n",
    "# 加载已保存的参数到克隆实例的隐藏层中。\n",
    "clone.hidden.load_state_dict(torch.load('mlp.hidden.params'))\n",
    "\n",
    "# 比较两个 MLP 实例的隐藏层权重是否相等，并输出结果。\n",
    "print(clone.hidden.weight == net.hidden.weight)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 练习 5.5.3\n",
    "\n",
    "如何同时保存网络架构和参数？需要对架构加上什么限制？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**解答：** "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;在PyTorch中，可以使用`torch.save()`函数同时保存网络架构和参数。为了保存网络架构，需要将模型的结构定义在一个Python类中，并将该类实例化为模型对象。此外，必须确保该类的构造函数不包含任何随机性质的操作，例如`dropout`层的随机丢弃率应该是固定的。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrderedDict([('hidden.weight',\n",
       "              tensor([[ 0.0369, -0.0737, -0.1655,  ..., -0.1467, -0.0981, -0.0649],\n",
       "                      [ 0.1146,  0.0826, -0.0368,  ..., -0.2168, -0.0170,  0.1297],\n",
       "                      [ 0.1910,  0.2167,  0.0187,  ..., -0.0515, -0.1264, -0.1181],\n",
       "                      ...,\n",
       "                      [-0.0703, -0.1536,  0.1264,  ..., -0.0806,  0.0988,  0.0529],\n",
       "                      [-0.2201,  0.0084, -0.0896,  ...,  0.0498, -0.2075, -0.1136],\n",
       "                      [-0.2109, -0.0641,  0.0418,  ..., -0.0203, -0.2172,  0.1407]])),\n",
       "             ('hidden.bias',\n",
       "              tensor([ 0.1411,  0.0751,  0.0969,  0.0029,  0.0372,  0.0706, -0.1384, -0.2168,\n",
       "                      -0.0420, -0.0457, -0.2219,  0.0156, -0.0376, -0.0149, -0.1756,  0.1775,\n",
       "                      -0.0498,  0.1740,  0.0502, -0.2122,  0.0721, -0.2034, -0.0786,  0.1569,\n",
       "                      -0.0462, -0.0403,  0.1635, -0.1923, -0.0664,  0.1024,  0.2127,  0.1872,\n",
       "                       0.0191, -0.1811, -0.0862, -0.2233,  0.0975, -0.0851,  0.0710, -0.1466,\n",
       "                      -0.2049,  0.1342, -0.1817,  0.0090, -0.0935, -0.1602,  0.1862, -0.1636,\n",
       "                      -0.0530,  0.2041, -0.0893, -0.2151,  0.0309, -0.2141,  0.2045, -0.1745,\n",
       "                       0.1725, -0.1740, -0.0616,  0.0855, -0.1436, -0.0203,  0.0422, -0.0495,\n",
       "                       0.0872, -0.1219, -0.1524,  0.1560,  0.1234, -0.2010,  0.0804,  0.0319,\n",
       "                       0.0674,  0.1892, -0.0612, -0.2194,  0.0423, -0.1021, -0.1069,  0.0444,\n",
       "                      -0.0778,  0.0706, -0.1959, -0.1350,  0.1476, -0.0056, -0.1770, -0.1335,\n",
       "                       0.0047,  0.0345,  0.1953, -0.0970,  0.1474, -0.1365, -0.0740, -0.1115,\n",
       "                       0.1277, -0.1293,  0.0805,  0.2143,  0.0489,  0.0239, -0.1202,  0.0630,\n",
       "                       0.0989, -0.1315, -0.2145, -0.0457,  0.1783,  0.1163,  0.0588, -0.2109,\n",
       "                       0.1293,  0.0125, -0.0506, -0.0688, -0.0362,  0.1525, -0.0767,  0.0250,\n",
       "                      -0.0215,  0.2067, -0.2087,  0.0019,  0.1837, -0.0515, -0.1267,  0.1714,\n",
       "                      -0.1128,  0.0493,  0.2092, -0.1758, -0.0071, -0.0011, -0.1364,  0.2054,\n",
       "                       0.1229, -0.1895,  0.1979,  0.1465, -0.1105,  0.1045, -0.0934, -0.0066,\n",
       "                       0.0594, -0.1511,  0.0321,  0.0790,  0.0740, -0.1771, -0.0988, -0.1070,\n",
       "                      -0.1435, -0.0270,  0.1004, -0.0928,  0.1074,  0.1110, -0.0949,  0.1733,\n",
       "                      -0.1990,  0.0028, -0.2005,  0.0088, -0.1635,  0.1489,  0.0688,  0.0156,\n",
       "                      -0.0585, -0.1188, -0.0556,  0.2090,  0.1175, -0.0224,  0.0818, -0.0736,\n",
       "                      -0.1864, -0.1766, -0.0220, -0.0968,  0.1442,  0.0663, -0.0458, -0.0430,\n",
       "                      -0.0526,  0.1705,  0.0098,  0.0074,  0.0626, -0.0427,  0.1415, -0.0658,\n",
       "                       0.1639, -0.0790,  0.0350,  0.1557,  0.0223,  0.1300,  0.1719, -0.1275,\n",
       "                      -0.0485, -0.0934,  0.1126,  0.1186, -0.0408,  0.2143,  0.1628, -0.1180,\n",
       "                      -0.0047, -0.0969,  0.0857,  0.2136, -0.1437,  0.0956, -0.0322,  0.1350,\n",
       "                       0.0727,  0.0507, -0.0803,  0.0739,  0.0788,  0.1012, -0.0323,  0.2109,\n",
       "                      -0.1115,  0.0882, -0.1487,  0.1048, -0.1192,  0.0686,  0.1347, -0.1369,\n",
       "                       0.1587, -0.0423, -0.0776, -0.0035,  0.0434,  0.0362, -0.1731, -0.1447,\n",
       "                       0.1940, -0.2227, -0.0927, -0.0368,  0.1222, -0.1020,  0.1389,  0.1392,\n",
       "                      -0.1096,  0.0886, -0.1027, -0.1677, -0.0990,  0.1158,  0.2233, -0.0059])),\n",
       "             ('output.weight',\n",
       "              tensor([[ 0.0192, -0.0188,  0.0375,  ...,  0.0599, -0.0187,  0.0173],\n",
       "                      [ 0.0553,  0.0517, -0.0476,  ..., -0.0349, -0.0505,  0.0260],\n",
       "                      [-0.0323, -0.0016, -0.0131,  ...,  0.0606, -0.0224, -0.0082],\n",
       "                      ...,\n",
       "                      [ 0.0172,  0.0622, -0.0192,  ..., -0.0166,  0.0531,  0.0336],\n",
       "                      [ 0.0423, -0.0094,  0.0205,  ..., -0.0002,  0.0353,  0.0409],\n",
       "                      [-0.0557,  0.0032,  0.0592,  ..., -0.0515, -0.0285,  0.0341]])),\n",
       "             ('output.bias',\n",
       "              tensor([ 0.0033,  0.0125, -0.0477,  0.0349,  0.0211, -0.0375,  0.0066,  0.0595,\n",
       "                       0.0455,  0.0521]))])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "from torch.nn import functional as F\n",
    "\n",
    "class MLP(nn.Module):\n",
    "    \"\"\"定义 MLP 类。\"\"\"             \n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.hidden = nn.Linear(20, 256)   # 定义隐藏层，输入尺寸为 20，输出尺寸为 256。\n",
    "        self.output = nn.Linear(256, 10)   # 定义输出层，输入尺寸为 256，输出尺寸为 10。\n",
    "\n",
    "    def forward(self, x):\n",
    "        \"\"\"定义前向传播函数。\"\"\"\n",
    "        # 使用 ReLU 激活函数，计算隐藏层和输出层的输出\n",
    "        return self.output(F.relu(self.hidden(x)))\n",
    "\n",
    "net = MLP()\n",
    "\n",
    "# 存储模型\n",
    "torch.save(net.state_dict(), 'model.pt')\n",
    "\n",
    "# 导入模型\n",
    "model = torch.load('model.pt')\n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.6 GPU "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 练习 5.6.1 \n",
    "\n",
    "尝试一个计算量更大的任务，比如大矩阵的乘法，看看CPU和GPU之间的速度差异。再试一个计算量很小的任务呢？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**解答：** "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;计算量很大的任务：使用GPU速度明显更快。\n",
    "\n",
    "&emsp;&emsp;计算量很小的任务：CPU速度可能更快，因为数据传输到GPU需要时间。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cpu time cost: 2317.36ms\n",
      "gpu time cost: 183.18ms\n",
      "cpu time cost: 1ms\n",
      "gpu time cost: 6ms\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "import torch\n",
    "\n",
    "# 计算量较大的任务\n",
    "X = torch.rand((10000, 10000))\n",
    "Y = X.cuda(0)\n",
    "time_start = time.time()\n",
    "Z = torch.mm(X, X)\n",
    "time_end = time.time()\n",
    "print(f'cpu time cost: {round((time_end - time_start) * 1000, 2)}ms')\n",
    "time_start = time.time()\n",
    "Z = torch.mm(Y, Y)\n",
    "time_end = time.time()\n",
    "print(f'gpu time cost: {round((time_end - time_start) * 1000, 2)}ms')\n",
    "\n",
    "# 计算量很小的任务\n",
    "X = torch.rand((100, 100))\n",
    "Y = X.cuda(0)\n",
    "time_start = time.time()\n",
    "Z = torch.mm(X, X)\n",
    "time_end = time.time()\n",
    "print(f'cpu time cost: {round((time_end - time_start) * 1000)}ms')\n",
    "time_start = time.time()\n",
    "Z = torch.mm(Y, Y)\n",
    "time_end = time.time()\n",
    "print(f'gpu time cost: {round((time_end - time_start) * 1000)}ms')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 练习 5.6.2\n",
    "\n",
    "我们应该如何在GPU上读写模型参数？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**解答：** "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;使用`net.to(device)`将模型迁移到GPU上，然后再按照之前的方法读写参数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrderedDict([('hidden.weight',\n",
       "              tensor([[ 0.0953, -0.1893,  0.0498,  ...,  0.1896, -0.1784, -0.0157],\n",
       "                      [-0.1258, -0.0917, -0.1756,  ..., -0.0070, -0.0495,  0.0975],\n",
       "                      [-0.1635, -0.0635,  0.1957,  ...,  0.0759, -0.0029, -0.1617],\n",
       "                      ...,\n",
       "                      [-0.1547,  0.1164, -0.1361,  ..., -0.1994,  0.0627, -0.0222],\n",
       "                      [ 0.1934,  0.1134,  0.0321,  ...,  0.0460, -0.0595, -0.1292],\n",
       "                      [ 0.0249,  0.1177,  0.0099,  ..., -0.0137,  0.1413, -0.1668]],\n",
       "                     device='cuda:0')),\n",
       "             ('hidden.bias',\n",
       "              tensor([-1.6038e-01, -2.1779e-01,  1.6563e-01, -8.3758e-02, -1.0277e-01,\n",
       "                      -4.7309e-02, -1.6957e-01, -2.0384e-01, -5.6247e-02,  1.6484e-01,\n",
       "                       1.8113e-02,  1.6802e-01, -6.7104e-02, -2.2222e-01,  1.5430e-01,\n",
       "                      -7.5118e-02,  7.3491e-04,  6.4319e-02,  7.5655e-02, -1.4435e-01,\n",
       "                       6.6799e-02,  3.6084e-02, -1.2386e-01, -1.3856e-01, -2.3952e-02,\n",
       "                      -1.0648e-02,  1.9898e-01, -1.9831e-01, -4.5019e-02, -1.0548e-01,\n",
       "                       9.5445e-02,  1.4266e-01,  6.1799e-02, -1.5684e-01, -1.2280e-01,\n",
       "                      -9.6995e-02, -1.1045e-01,  1.3525e-01, -1.2214e-01,  1.0288e-01,\n",
       "                       3.7759e-02,  1.7944e-01, -7.7941e-02, -9.1030e-02, -9.5729e-02,\n",
       "                       1.3995e-01,  1.8401e-01,  8.6624e-02, -2.0561e-01,  3.9184e-02,\n",
       "                      -7.1068e-02,  2.3956e-02,  9.2729e-02, -5.6481e-02, -3.1340e-02,\n",
       "                       1.4743e-01,  1.3637e-01, -1.2101e-01, -9.7500e-02, -1.9022e-02,\n",
       "                       9.2734e-02,  6.2800e-02,  2.1892e-01, -7.3675e-02, -4.1030e-02,\n",
       "                       1.9740e-01, -6.3016e-03, -1.5521e-01, -7.8558e-02, -8.7364e-02,\n",
       "                      -7.9161e-02,  1.5430e-01,  1.8954e-02, -1.9178e-01,  1.9881e-01,\n",
       "                       2.2027e-01, -1.8078e-01,  6.9323e-02, -4.6385e-02,  2.1702e-01,\n",
       "                       1.4624e-01, -8.3725e-02,  7.0453e-02,  1.9612e-01,  3.6836e-02,\n",
       "                       2.2098e-01,  3.0478e-02, -4.7391e-02, -1.4902e-01,  7.9605e-02,\n",
       "                       1.9745e-01, -6.4884e-02,  1.6423e-01,  1.2921e-01,  3.0099e-02,\n",
       "                       1.5715e-01,  1.5617e-01,  1.3073e-01,  1.1184e-01,  1.3289e-01,\n",
       "                       1.2198e-01, -4.0842e-02,  1.1024e-01, -1.5366e-01, -1.9910e-02,\n",
       "                       9.2563e-02,  5.6079e-02, -2.7985e-02,  2.1201e-01, -1.1703e-01,\n",
       "                       5.8150e-02, -1.1542e-01, -2.1905e-01, -1.8184e-01,  2.6795e-02,\n",
       "                      -1.2731e-01,  6.0692e-02,  5.5184e-02, -8.2398e-02, -4.8447e-02,\n",
       "                      -8.1548e-02,  7.0197e-02, -1.5017e-01, -5.0244e-02,  1.1265e-01,\n",
       "                       1.7183e-01,  2.8989e-03,  1.9149e-04,  3.3335e-02,  1.2014e-01,\n",
       "                       9.2528e-02, -1.2463e-01, -1.1867e-01,  3.7557e-02,  1.5263e-01,\n",
       "                       8.6338e-02,  8.7072e-02, -7.8383e-02, -1.7524e-01, -6.7949e-02,\n",
       "                       2.0034e-01, -7.7246e-02,  1.0781e-01,  1.6412e-02,  8.1597e-02,\n",
       "                       8.4905e-02, -2.1820e-01, -3.9792e-02, -9.9967e-02,  2.6758e-03,\n",
       "                      -1.4489e-01,  5.5969e-03,  1.9850e-01, -7.2962e-02, -1.2461e-01,\n",
       "                       7.4121e-02, -1.1330e-01, -1.4070e-01, -1.5462e-01,  6.8374e-02,\n",
       "                       1.2733e-01,  6.0084e-03, -7.6969e-02,  5.7821e-02, -1.8573e-01,\n",
       "                       2.1110e-01, -3.0443e-02,  6.4360e-02, -1.2431e-01,  2.0503e-01,\n",
       "                      -1.4694e-01, -2.5405e-02,  7.3424e-02,  1.1856e-02, -3.3563e-02,\n",
       "                      -6.4873e-02, -1.3615e-01, -9.5351e-02,  1.5496e-01,  1.6950e-01,\n",
       "                      -9.6969e-02, -3.0892e-02,  1.6642e-01, -1.3740e-01,  2.9468e-02,\n",
       "                       2.2358e-01,  1.8460e-01,  3.6118e-02, -5.1945e-02, -1.0869e-01,\n",
       "                      -1.9553e-01,  1.3739e-01,  1.4731e-01, -8.9318e-02, -4.6817e-02,\n",
       "                       4.0572e-02, -1.4156e-01,  4.7386e-03, -2.6578e-02,  1.0131e-01,\n",
       "                      -2.0924e-01, -4.4850e-02,  2.0484e-02, -1.9844e-01, -2.1546e-01,\n",
       "                      -2.9280e-02,  7.5573e-02,  7.3782e-02,  1.8705e-01, -1.1650e-01,\n",
       "                      -4.4113e-02,  1.7739e-01,  1.0117e-01, -1.5911e-01, -1.1847e-01,\n",
       "                       1.0833e-02,  3.7645e-02,  4.6225e-02, -3.5639e-02,  2.5980e-02,\n",
       "                       1.4862e-01,  1.8359e-01,  7.7204e-02,  2.0624e-01, -1.1387e-02,\n",
       "                       1.8623e-01, -3.3906e-02,  8.5860e-02, -3.0297e-02, -1.2838e-01,\n",
       "                      -1.7408e-02,  1.3664e-01, -2.8860e-02, -7.6724e-02,  1.6023e-01,\n",
       "                      -3.4478e-02, -4.8468e-02, -1.2971e-01,  1.5275e-01,  1.1766e-01,\n",
       "                      -3.9613e-02, -2.9950e-03, -1.0539e-01,  1.4344e-01, -2.0152e-01,\n",
       "                      -2.1304e-02, -1.2966e-01, -1.0855e-01, -3.7007e-02, -2.2111e-01,\n",
       "                      -9.8307e-02,  4.1741e-02, -1.6946e-01,  1.2277e-01,  1.0471e-01,\n",
       "                      -1.5905e-01], device='cuda:0')),\n",
       "             ('output.weight',\n",
       "              tensor([[-0.0487, -0.0548, -0.0404,  ...,  0.0182, -0.0258,  0.0312],\n",
       "                      [ 0.0209,  0.0136, -0.0246,  ..., -0.0532, -0.0306, -0.0233],\n",
       "                      [ 0.0279,  0.0110,  0.0500,  ..., -0.0120,  0.0273, -0.0516],\n",
       "                      ...,\n",
       "                      [ 0.0444, -0.0426,  0.0023,  ..., -0.0032,  0.0435, -0.0229],\n",
       "                      [ 0.0609,  0.0273,  0.0505,  ...,  0.0355,  0.0315,  0.0373],\n",
       "                      [ 0.0553,  0.0520, -0.0298,  ...,  0.0266, -0.0455,  0.0360]],\n",
       "                     device='cuda:0')),\n",
       "             ('output.bias',\n",
       "              tensor([-0.0039, -0.0523,  0.0351,  0.0334, -0.0173, -0.0601,  0.0499, -0.0320,\n",
       "                       0.0240,  0.0089], device='cuda:0'))])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "from torch import nn\n",
    "from torch.nn import functional as F\n",
    "\n",
    "class MLP(nn.Module):\n",
    "    \"\"\"定义 MLP 类。\"\"\"\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.hidden = nn.Linear(20, 256)   # 定义隐藏层，输入尺寸为 20，输出尺寸为 256。\n",
    "        self.output = nn.Linear(256, 10)   # 定义输出层，输入尺寸为 256，输出尺寸为 10。\n",
    "\n",
    "    def forward(self, x):\n",
    "        \"\"\"定义前向传播函数。\"\"\"\n",
    "        # 使用 ReLU 激活函数，计算隐藏层和输出层的输出。\n",
    "        return self.output(F.relu(self.hidden(x)))  \n",
    "\n",
    "# 选择GPU，没有GPU就选CPU\n",
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "# 创建模型实例对象\n",
    "net = MLP()\n",
    "# 将模型参数传输到GPU上\n",
    "net.to(device)\n",
    "# 访问模型参数\n",
    "net.state_dict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 练习 5.6.3 \n",
    "\n",
    "测量计算1000个$100 \\times 100$矩阵的矩阵乘法所需的时间，并记录输出矩阵的Frobenius范数，一次记录一个结果，而不是在GPU上保存日志并仅传输最终结果。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**解答:** "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;中文版翻译有点问题，英文原版这句话是：\n",
    "\n",
    ">Measure the time it takes to compute 1000 matrix-matrix multiplications of $100×100$ matrices and log the Frobenius norm of the output matrix one result at a time vs. keeping a log on the GPU and transferring only the final result.\n",
    "\n",
    "&emsp;&emsp;所以这道题的本质还是希望我们做个比较。\n",
    "\n",
    "&emsp;&emsp;实验一：仅记录1000次$100×100$矩阵相乘所用的时间，不需要打印Frobenius范数。\n",
    "\n",
    "&emsp;&emsp;实验二：记录1000次$100×100$矩阵相乘所用的时间，并打印Frobenius范数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time taken: 0.11336207389831543\n",
      "Time taken: 0.0427088737487793\n",
      "实验一消耗时间：0.11336207389831543，实验二消耗时间：0.0427088737487793\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import time\n",
    "\n",
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "# 生成随机矩阵\n",
    "matrices = [torch.randn(100, 100).to(device) for i in range(1000)]\n",
    "\n",
    "# 实验一：计算时间\n",
    "start_time_1 = time.time()\n",
    "for i in range(1000):\n",
    "    result = torch.mm(matrices[i], matrices[i].t())\n",
    "    frobenius_norm = torch.norm(result)\n",
    "#     print(frobenius_norm)\n",
    "end_time_1 = time.time()\n",
    "print(\"Time taken:\", end_time_1 - start_time_1)\n",
    "\n",
    "# 实验二：计算时间\n",
    "start_time_2 = time.time()\n",
    "for i in range(1000):\n",
    "    result = torch.mm(matrices[i], matrices[i].t())\n",
    "    frobenius_norm = torch.norm(result)\n",
    "    # print(frobenius_norm)\n",
    "end_time_2 = time.time()\n",
    "print(\"Time taken:\", end_time_2 - start_time_2)\n",
    "\n",
    "print(f'实验一消耗时间：{end_time_1 - start_time_1}，实验二消耗时间：{end_time_2 - start_time_2}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 练习 5.6.4\n",
    "\n",
    "测量同时在两个GPU上执行两个矩阵乘法与在一个GPU上按顺序执行两个矩阵乘法所需的时间。提示：应该看到近乎线性的缩放。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**解答：** "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "&emsp;&emsp;执行两个矩阵乘法并行在两个GPU上所需的时间通常会比在单个GPU上按顺序执行这两个操作要快得多。但实际的时间取决于矩阵的大小、硬件配置和算法实现。\n",
    "\n",
    "&emsp;&emsp;但由于笔者只有一张卡，所以只做了在单个GPU上顺序执行两个矩阵乘法的实验。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sequential time: 0.00350714 seconds\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "import time\n",
    "\n",
    "# 创建两个随机矩阵\n",
    "a = torch.randn(10000, 10000).cuda()\n",
    "b = torch.randn(10000, 10000).cuda()\n",
    "\n",
    "# 顺序计算\n",
    "start_time = time.time()\n",
    "c1 = torch.matmul(a, b)\n",
    "c2 = torch.matmul(a, b)\n",
    "end_time = time.time()\n",
    "sequential_time = end_time - start_time\n",
    "\n",
    "print(f\"Sequential time: {sequential_time:.8f} seconds\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
